Telemetry for OTLP traces/metrics/logs by mcculls · Pull Request #12057 · DataDog/dd-trace-java

mcculls · 2026-07-23T14:35:05Z

What Does This Do

Telemetry for OTLP traces/metrics/logs:

otel.traces_export_{attempts,successes,failures}
otel.metrics_export_{attempts,successes,failures}
otel.log_records

OTLP telemetry metrics are tagged by protocol and encoding (per signal)

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

mcculls · 2026-07-23T14:35:19Z

@codex review

Copilot

Pull request overview

This PR adds telemetry coverage for OTLP-based exporting across traces, metrics, and logs in dd-trace-java, by emitting agent telemetry metrics for OTLP exporter activity and by wiring OTLP trace sends into existing Datadog tracer health metrics reporting.

Changes:

Add a new OtlpTelemetry metric collector (internal-api) and a telemetry periodic action (telemetry) to report OTLP metrics/logs telemetry metrics with protocol/encoding tags.
Refactor OTLP senders (OtlpSender) to return RemoteApi.Response, centralizing HTTP execution/handling in OtlpSenderSupport, and use responses to drive success/failure counters.
Feed OTLP trace send results into HealthMetrics via OtlpPayloadDispatcher (used by OtlpWriter), and update tests for the new sender contract.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
telemetry/src/main/java/datadog/telemetry/TelemetrySystem.java	Registers the new OTLP telemetry periodic action in the telemetry loop.
telemetry/src/main/java/datadog/telemetry/metric/OtlpTelemetryPeriodicAction.java	Adds a `MetricPeriodicAction` to drain OTLP telemetry metrics into telemetry.
internal-api/src/main/java/datadog/trace/api/telemetry/OtlpTelemetry.java	Implements OTLP exporter telemetry metrics (metrics export counters + log record count) tagged by protocol/encoding.
internal-api/src/test/java/datadog/trace/api/telemetry/OtlpTelemetryTest.java	Adds unit tests validating metric names, tags, draining behavior, and count handling.
dd-trace-core/src/main/java/datadog/trace/core/otlp/common/OtlpSender.java	Changes sender contract to return `RemoteApi.Response` so callers can track success/failure.
dd-trace-core/src/main/java/datadog/trace/core/otlp/common/OtlpSenderSupport.java	New shared helper for executing OTLP requests with retries and mapping outcomes to `RemoteApi.Response`.
dd-trace-core/src/main/java/datadog/trace/core/otlp/common/OtlpHttpSender.java	Uses `OtlpSenderSupport` and returns `RemoteApi.Response`.
dd-trace-core/src/main/java/datadog/trace/core/otlp/common/OtlpGrpcSender.java	Uses `OtlpSenderSupport` and returns `RemoteApi.Response`.
dd-trace-core/src/main/java/datadog/trace/core/otlp/metrics/OtlpMetricsService.java	Records OTLP metrics export attempts/successes/failures to `OtlpTelemetry` based on response.
dd-trace-core/src/main/java/datadog/trace/core/otlp/logs/OtlpLogsCollector.java	Extends collector API to expose the number of log records collected.
dd-trace-core/src/main/java/datadog/trace/core/otlp/logs/OtlpLogsProtoCollector.java	Tracks log record count during protobuf collection.
dd-trace-core/src/main/java/datadog/trace/core/otlp/logs/OtlpLogsJsonCollector.java	Tracks log record count during JSON collection.
dd-trace-core/src/main/java/datadog/trace/core/otlp/logs/OtlpLogsService.java	Emits `otel.log_records` telemetry on successful OTLP log export.
dd-trace-core/src/main/java/datadog/trace/common/writer/OtlpWriter.java	Passes `HealthMetrics` into the OTLP payload dispatcher.
dd-trace-core/src/main/java/datadog/trace/common/writer/OtlpPayloadDispatcher.java	Reports OTLP trace send success/failure to `HealthMetrics` using `RemoteApi.Response`.
dd-trace-core/src/test/java/datadog/trace/common/writer/OtlpPayloadDispatcherTest.java	Updates mocks/stubbing for new `OtlpSender.send` return type.
dd-trace-core/src/test/java/datadog/trace/core/otlp/metrics/OtlpStatsMetricWriterTest.java	Updates test sender to satisfy new `OtlpSender.send` signature.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 62f5b3cdb7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

dd-octo-sts · 2026-07-23T14:57:02Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.94 s	13.86 s	[-0.1%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent	12.87 s	12.97 s	[-1.5%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent	16.91 s	16.52 s	[+1.6%; +3.2%] (significantly worse)
startup:petclinic:iast:Agent	16.86 s	16.97 s	[-1.5%; +0.2%] (no difference)
startup:petclinic:profiling:Agent	16.73 s	16.77 s	[-1.4%; +0.9%] (no difference)
startup:petclinic:sca:Agent	17.00 s	16.85 s	[-0.1%; +1.8%] (no difference)
startup:petclinic:tracing:Agent	16.02 s	16.26 s	[-2.4%; -0.6%] (maybe better)

Commit: 66cce163 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

datadog-datadog-prod-us1-2

More details

The diff preserves the existing OTLP retry and payload paths while adding counters that reset at collection boundaries and only count successfully exported traces or log records. Focused Gradle execution was blocked before compilation because the environment lacks the repository-required Java 25 toolchain; no concrete behavioral regression was identifiable from the diff and surrounding code.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 839240f · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 839240f710

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

* Datadog tracer health metrics * otel.metrics_export_{attempts,successes,failures} * otel.log_records OTLP telemetry metrics are tagged by protocol and encoding (per signal) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

mhlidd

Small nit about Log metrics, otherwise LGTM

…tatsD client) and use telemetry instead

mcculls · 2026-07-23T22:19:01Z

@codex review

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

dd-trace-core/src/main/java/datadog/trace/common/writer/WriterFactory.java:82

In the OTLP writer branch, WriterFactory.createWriter(...) no longer passes its healthMetrics argument into the OTLP writer builder. This means enabling tracer health metrics has no effect for OTLP, unlike other writer types handled in this factory.

If OTLP writers are still expected to emit the standard tracer health metrics, the OTLP builder should accept HealthMetrics and this factory should forward the healthMetrics instance here.

    } else if (OTLP_WRITER_TYPE.equals(configuredType)) {
      return OtlpWriter.builder()
          .endpoint(config.getOtlpTracesEndpoint())
          .headers(config.getOtlpTracesHeaders())
          .protocol(config.getOtlpTracesProtocol())
          .compression(config.getOtlpTracesCompression())
          .timeoutMillis(config.getOtlpTracesTimeout())
          .spanSamplingRules(singleSpanSampler)
          .flushIntervalMilliseconds(flushIntervalMilliseconds)
          .build();
    }

internal-api/src/test/java/datadog/trace/api/telemetry/OtlpTelemetryTest.java:66

This test hard-codes expected OTLP protocol/encoding tags (protocol:http, encoding:protobuf). Since the tags in OtlpTelemetry are derived from Config.get().getOtlpMetricsProtocol(), assert against the protocol-derived values to avoid test-order dependence on system properties.

    assertTrue(metric.tags.contains("protocol:http"));
    assertTrue(metric.tags.contains("encoding:protobuf"));

internal-api/src/test/java/datadog/trace/api/telemetry/OtlpTelemetryTest.java:95

This test assumes logs exporter tags are always protocol:http and encoding:protobuf, but OtlpTelemetry derives them from Config.get().getOtlpLogsProtocol(). Computing the expected tag values from the configured protocol will keep the test stable if other tests set OTLP-related system properties.

    assertTrue(metric.tags.contains("protocol:http"));
    assertTrue(metric.tags.contains("encoding:protobuf"));

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f446803c0f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

mcculls · 2026-07-23T22:55:53Z

@codex review

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

dd-trace-core/src/test/java/datadog/trace/common/writer/OtlpPayloadDispatcherTest.java:144

OtlpTelemetry.drain() does not include newly incremented counters unless prepareMetrics() has been called to stage them into the queue. As written, this helper will always return an empty map, making the telemetry assertions fail.

Stage metrics via prepareMetrics() before draining.

  private static Map<String, OtlpTelemetry.OtlpMetric> drainTracesTelemetry() {
    Map<String, OtlpTelemetry.OtlpMetric> byName = new HashMap<>();
    for (OtlpTelemetry.OtlpMetric metric : OtlpTelemetry.getInstance().drain()) {
      byName.put(metric.metricName, metric);
    }

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1a0ba1d4dd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

mcculls added type: feature Enhancements and improvements inst: opentelemetry OpenTelemetry instrumentation labels Jul 23, 2026

mcculls requested a review from Copilot July 23, 2026 14:35

Copilot started reviewing on behalf of mcculls July 23, 2026 14:35 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/otlp/logs/OtlpLogsService.java

Comment thread dd-trace-core/src/main/java/datadog/trace/common/writer/OtlpPayloadDispatcher.java Outdated

Comment thread dd-trace-core/src/main/java/datadog/trace/core/otlp/common/OtlpSenderSupport.java

This comment has been minimized.

Sign in to view

mcculls force-pushed the mcculls/otlp-telemetry-metrics branch 2 times, most recently from ab35f2c to 839240f Compare July 23, 2026 16:36

mcculls marked this pull request as ready for review July 23, 2026 16:49

mcculls requested review from a team as code owners July 23, 2026 16:49

mcculls requested review from amarziali and removed request for a team July 23, 2026 16:49

dd-octo-sts Bot added the tag: ai generated Largely based on code generated by an AI or LLM label Jul 23, 2026

mcculls requested a review from mhlidd July 23, 2026 16:49

datadog-datadog-prod-us1-2 Bot reviewed Jul 23, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/otlp/metrics/OtlpMetricsService.java Outdated

Comment thread dd-trace-core/src/main/java/datadog/trace/common/writer/OtlpPayloadDispatcher.java Outdated

mcculls force-pushed the mcculls/otlp-telemetry-metrics branch from 839240f to 12930e9 Compare July 23, 2026 17:30

Telemetry for OTLP traces/metrics/logs:

74e91ad

* Datadog tracer health metrics * otel.metrics_export_{attempts,successes,failures} * otel.log_records OTLP telemetry metrics are tagged by protocol and encoding (per signal) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

mcculls force-pushed the mcculls/otlp-telemetry-metrics branch from 12930e9 to 74e91ad Compare July 23, 2026 17:34

mhlidd approved these changes Jul 23, 2026

View reviewed changes

Comment thread internal-api/src/main/java/datadog/trace/api/telemetry/OtlpTelemetry.java

mcculls added 2 commits July 23, 2026 23:03

Remove HealthMetrics for OTLP traces (since HealthMetrics relies on S…

91c11f7

…tatsD client) and use telemetry instead

Simplify OTLP telemetry

f446803

mcculls requested a review from Copilot July 23, 2026 22:18

Copilot started reviewing on behalf of mcculls July 23, 2026 22:19 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/common/writer/OtlpWriter.java

Comment thread internal-api/src/test/java/datadog/trace/api/telemetry/OtlpTelemetryTest.java

chatgpt-codex-connector Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/common/writer/OtlpWriter.java

Comment thread internal-api/src/main/java/datadog/trace/api/telemetry/OtlpTelemetry.java Outdated

mcculls requested a review from Copilot July 23, 2026 22:55

Copilot started reviewing on behalf of mcculls July 23, 2026 22:56 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread dd-trace-core/src/test/java/datadog/trace/common/writer/OtlpPayloadDispatcherTest.java

Stage telemetry metrics separate to collection

66cce16

mcculls force-pushed the mcculls/otlp-telemetry-metrics branch from 1a0ba1d to 66cce16 Compare July 23, 2026 23:03

chatgpt-codex-connector Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread dd-trace-core/src/test/java/datadog/trace/common/writer/OtlpPayloadDispatcherTest.java

Comment thread dd-trace-core/src/main/java/datadog/trace/core/otlp/common/OtlpSenderSupport.java

Uh oh!

Conversation

mcculls commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Contributor Checklist

Uh oh!

mcculls commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

mhlidd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mcculls commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

mcculls commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mcculls commented Jul 23, 2026 •

edited

Loading

dd-octo-sts Bot commented Jul 23, 2026 •

edited

Loading